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Early  cost  estimation  methods  often  resuit  in  highiy  inaccurate 
program  cost  predictions  -  and  it  continues  to  worsen 


Table  1:  Analysis  of  DOD  Major  Defense  Acquisition  Program  Portfolios 


Fiscal  year  2008  dollars 


Fiscal  year 


2000  portfolio  2005  portfolio  2007  portfolio 


Portfolio  size 


Number  of  programs 

75 

91 

QS 

Total  planned  commitments 

1790  Billion 

$1 .5  Trillion 

$1 .8  Trillion 

Commitments  outstanding 

$380  Billion 

$887  Billion 

$858  Billion 

Portfolio  performance 

r 

Change  to  total  RDT&E  costs 
from  first  estimate 

27  percent 

38  percent 

40  peircent 

Change  in  totai  acquisition  cost 
from  first  estimate 

8  percent 

1 8  percent 

26  percent 

Estimated  totai  acquisition  cost 
growth 

^  $42  Billion 

$202  Billion 

S29G  Billion 

Share  of  programs  with  2E 
percent  or  more  increase  in 
program  acquisition  unit  cost 

37  percent 

44  percent 

44  fjercent 

Average  schedule  delay  in 
delivering  initial  capabilities 


16  months 


17  months 


21  months 


Unsustainable 
negative  trend 
in  cost 
predictions 


Scunce:  GAO  DDD  satB. 

Source:  Fundamental  Changes  Are  Needed  to  Improve  Weapon  Program  Outcomes,  GAO  Testimony  Before  the  Subcommittee  on  Federal 
Financial  Management,  Government  Information,  Federal  Services,  and  International  Security,  Committee  on  Homeland  Security  and 
Governmental  Affairs,  U.S.  Senate,  Sept  25,  2008  GAO-08-1159T 
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“DOD’s  flawed  funding  process  is  largely  driven  by  decision  makers’  willingness  to  accept 
unrealistic  cost  estimates  and  DOD’s  commitment  to  more  programs  than  it  can  support.  DOD 
often  underestimates  deveiopment  costs — due  in  part  to  a  iack  of  knowiedge  and  optimistic 
assumptions  about  requirements  and  criticai  technoiogies.”  * 


Funding  Shortfallii  at  ttie  Start  of  Development  for  Five  Mafor  Weapon  Sye^em  Programs 

Program 


Muhi-mission 
Maritime  Aircraft 
Warfighlar  JivTarmatlon 
Helwork-TaDtical 

Future  Combat  Systems 
Joint  Strike  Fighter 
Global  Hawk 


0  10  20  30  40  50  60  70  BO  90  100 

Percantage  ol  development  funding 


Lwel  of  funding  established  n  the  FYDP  in  the  year  pro^am  was  initiated 
I  I  Lwel  of  funding  the  program  needed  to  be  fully  funded  in  the  ntial  FYDP 

I  I  Funding  required  beyond  the  iiitiai  FYDP  Id  compleledevefepmerYt 

Source:  DOD  (data|:  OAO  (analysis  and  pnesertaUonj. 


*Source:  A  Knowledge-Based  Funding  Approach  Could  Improve  Major  Weapon  System  Program  Outcomes,  GAO 
Report  to  the  Committee  on  Armed  Services,  U.S.  Senate  s,  U.S.  Senate,  July,  2008  GAO-08-619 
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Real  cost  overruns  (in  2009  $  billions) 


Functional  reasons  for  cost  overruns 


$200 
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Cost  overruns 
123% 


$115 1 


IA6% 

3%  $41 13% 


15% 

13% 
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$40  p  3% 

$3  ^0.4% 

-$3 
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-$13 


-$2 


-$71 


$0 


0.3% 


Cost  savings 


30% 


15% 


0% 


-15% 


Quantity  Schedule  Engineering  f  Estimating  I  Support  Other 

I  Cost  overrun  ■  Cost  decrease  •Average  overrun  ♦  Baseline-weighted  average  overrun 


Source:  December  2009  SAR;  analysis  by  CSIS  Defense-Industrial  Initiatives  Group 

Cost  and  Time  Overruns  for  Major  Defense  Acquisition  Programs,  2010 
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Real  cost  overruns  {in  percent) 


DoD  Acquisition  Lifecycie 


Acquisition  Phases  and  Decision  Milestones 


Materiei 

Soiution 


ATechnoiogy 
Development 


A  Engineering 
&  Manufacturing 


Production 
&  Deployment 


Cost  Estimate 

Based  on: 

•  Analogies 

•  Expert  Judgment 

•  Limited  Information 


$ 


N 


Delay 


Ground  Combat  Vehicle  Delay  Due  to 
Reconciling  Cost  Estimates 

•  4  months  delay  in  obtaining  approval  to  proceed 

•  Rework  to  conduct  a  new  Analysis  of 
Alternatives  and  to  produce  a  new  cost  estimate 

Source:  GAO-1 2-1 81 T 


Cost  Growth 


PCS  Program  2003  vs  2009 

•  Status  -  program  terminated 

•  Cost  estimate  grew  by  $70B 

•  Schedule  grew  from  7.5  to  12.3  yrs 

•  Lines  of  code  grew  from  34M  to 
114M 

Source:  GAO-1 0-406 
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Expert  Judgements 


Information  Flow  for  Early  Lifecycle  Estimation 


Information  from  Analogous  Programs/Systems 
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Proposed  Material  Solution  &  Analysis  of  Alternatives 


Program  Execution  Change  Drivers 


System  Characteristics 

Trade-offs 


Operational  Capability 

Trade-offs 
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Create  a  Method  for  Quantifying  the  Uncertainty  of  Cost 
Estimation  inputs  and  Resuiting  Estimates 


1.  Identify  Change 
Drivers  &  States 


Elements  of  Innovation 

Explicit  identification  of  domain  specific  program  change  drivers. 


2.  Reduce  Cause  and 
Effect  Relationships 
via  Dependency 
Structure  Matrix 
techniques 


3.  Assign  Conditional 
Probabilities  to  BBN 
Model 


4.  Calculate  Cost 
Factor  Distributions 
for  Program 
Execution  Scenarios 


5.  Monte  Carlo 
Simulation  to 
Compute  Cost 
Distribution 


Unique  application  of  Dependency  Structure  Matrix  techniques  for  cost 
estimation. 


BBN  modeling  of  a  larger  number  of  program  change  drivers  for 
estimation  than  previous  research. 


Scenario  modeling  of  alternate  program  executions  to  assess  influence  of 
various  underlying  assumptions. 

Monte  Carlo  simulation  applied  to  estimation  input  parameters  rather  than 
output  values. 


Technical  Problem 

Complexity  Reduction 

Modeling  Uncertainty 
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step  1 :  Identify  Change  Drivers  and  States 


step  2:  Reduce  Cause  and  Effect  Relationships  via 
Design  Structure  Matrix  Techniques  y. 

Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate  /A 


Closing  Techi 

Building  Tech  Capturing  interrelationships  among  change  drivers  and 

Interoperabilil 

Systems  Des  rsducing  the  complexity  of  the  network 
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step  2:  Reduce  Cause  and  Effect  Relationships  via 
Dependency  Structure  Matrix  Techniques 


Change  Drivers  -  Cause  &  Effects  Matrix 


Mission  /  CONORS 
Change  in  Strategic  Vision 
Capability  Definition 
Advocacy  Change 
Closing  Technical  Gaps  (CBA) 

Building  Technical  Capability  &  Capacity  (CBA) 

Interoperability 

Systems  Design 

Interdependency 

Functional  Measures 

Scope  Definition 

Functional  Solution  Criteria  (measure) 

Funding  Schedule 

Acquisition  Management 

Program  Mgt  -  Contractor  Relations 

Project  Social  /  Dev  Env 

Prog  Mgt  Structure 

Manning  at  program  office 

Scope  Responsibility 

S  ta  nd  a  rd  s/ Ce  rt  if  i  cat  i  o  ns 

Supply  Chain  Vulnerabilities 

Information  sharing 

PO  Process  Performance 

Sustainment  Issues 

Contract  Award 

Production  Quantity 

Data  Ownership 

Industry  Company  Assessment 

Cost  Estimate 

Test  &  Evaluation 

Contractor  Performance 

Size 

Project  Challenge 
Product  Challenge 

Totals  0 

Below  diagonal _ 0 


5  12 
4  4 
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step  3:  Assign  Conditional  Probabilities  to  BBN  Model 


Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


1.  Identify 
Change 
Drivers  & 
States 


2.  Reduce  Cause  and 
Effect  Reiationships  via 
Dependency  Structure 
Matrix  techniques 


3.  Assign 
Conditionai 
Probabiiities  to 
BBN  Modei 


I 


4.  Caicuiate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 


5.  Monte  Cario 
Simuiation  to 
Compute  Cost 
Distribution 


!  Capability  Definition 


Node  Probability  Table 


Capability  Definition 
is  affected  by 
CONORS  and 
Strategic  Vision 
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Manual 


Quantifying  the 
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drivers  and  the 
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step  4:  Calculate  Cost  Factor  Distributions  for  Program 
Execution  Scenarios  ^ 

Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


1.  Identify 
Change 
Drivers  & 
States 


2.  Reduce  Cause  and 
Effect  Reiationships  via 
Dependency  Structure 
Matrix  techniques 


3.  Assign 
Conditionai 
Probabiiities  to 
BBN  Modei 


4.  Caicuiate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 


5.  Monte  Cario 
Simuiation  to 
Compute  Cost 
Distribution 


Program  Mgt 
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n  4% 

— IScenario  1 
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23% 
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3.0  -  4.0 


An  example 
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drivers  in  nominal 
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BBN  model  enables  computation  of  different  scenarios  of  — 
program  execution  on  cost  model  factors 
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step  5a:  Monte  Carlo  Simulation  to  Compute  Cost  Distribution 


Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


1.  Identify 
Change 
Drivers  & 
States 


2.  Reduce  Cause  and 
Effect  Reiationships  via 
Dependency  Structure 
Matrix  techniques 


3.  Assign 
Conditionai 
Probabiiities  to 
BBN  Modei 


4.  Caicuiate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 


5.  Monte  Cario 
Simuiation  to 
Compute  Cost 
Distribution 


BBN  output  distributions  mapped  to 
COCOMO  input  values 


n*  Assumption:  Cell  B3 
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Custom  Distribution 
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uncertainty  of  program  change  drivers  to  cost  drivers 
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COCOMO  “Architecture”  Parameter  Mapping 


Product  challenge  factors  represent  uncertainty  in  performance  criteria 
and  technology. 

PREC:  Is  this  application  unprecedented? 

FLEX:  How  stringent  are  the  product  goals,  scope  and  objectives? 

RCPX:  What  is  required  product  reliability  and  complexity? 

RUSE:  Must  we  design  for  re-usability? 

PDIF:  Platform  difficulty?  Processing  speed,  memory?  Platform  stability? 
RESL:  Have  we  addressed  technology  &  architecture  risk? 

Project  challenge  factors  represent  difficulty  in  managing  the  workforce. 
PREX:  Personnel  capability  and  experience? 

SCED:  How  much  schedule  pressure  is  applied  to  this  development? 

FOIL:  Are  facilities  adequate?  Includes  tools  and  multi-site  development. 
TEAM:  Do  we  have  a  cohesive  development  team? 

PMAT:  Does  the  organization  have  a  mature  process? 


QUELCE 
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step  5b:  Monte  Carlo  Simulation  to  Compute  Cost  Distribution 


Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


A 

1  Effect 

2  Product  Challenge 

3  Project  Challenge 

4  Estimated  Size  (KSLOC) 

5  Product  Challenge  factors 

6  COCOMO  Parameter 

7  Scale  Factors 
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BBN  Outputs 


50 
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k  I -Infinity 


Certainty:  [90.0000 


i  1 1,854.48 
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Monte  Carlo  Simulation 


We  will  use  Monte  Carlo  simulation  to  connect  the  BBN  output  node 
distributions  to  the  COCOMO  input  parameter  distributions 


The  animation  on  the  next  slide  depicts  the  essence  of  Monte  Carlo 
simulation  when  we  need  to  work  with  distributions  rather  than  single 
numbers 
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2  3 


1  2  3  4  5 

Crystal  Ball  then 
allows  the  user  to 
analyze  and  interpret 
the  final  distribution  of 
C! 


Crystai  Baii  uses  a 
random  number 
generator  to  seiect 
vaiues  for  A  and  B 


B 


123456789  10 
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Crystal  Ball  causes 
Excel  to  recalculate  all 
cells,  and  then  it 
saves  off  the  different 
results  for  C! 
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An  Example  Output  of  Monte  Carlo  Simulation 


200,000  Trials 


Frequency  View 


199,650  Displayed 
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Develop  Efficient  Techniques  To  Caiibrate  Expert 
Judgment  of  Program  Uncertainties 


Solution 


Step  2:  Iterate 
through  a  series 
of  domain 
specific  tests 


Step  1 :  Virtual 
training  using 
reference 
points 
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Step  3:  Feedback  on 
test  performance 
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Outcome:  Expert 
renders  calibrated 
estimate  of  size 


DoD  Domain-Specific 
reference  points 


1) 


2) 


3) 


Size  of  ground  combat  vehicie 
targeting  feature  xyz  in  2002 
consisted  of  25  KSLOC  Ada 

Size  of  Army  artiiiery  firing 
capabiiity  feature  abc  in  2007 
consisted  of  18  KSLOC  C++ 
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90%  1 00% 


Assessed  Chance  Of  Being  Correct 


Calibrated  =  more 
realistic  size  and 
wider  range  to 
reflect  true  expert 
uncertainty 


Estimate  of  SW  Size 


Used  with  permission  from  Douglas  Hubbard  Copyright  HDR  2008  dwhubbard@hubbardresearch.com 
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Polling  Question  1 

Do  you  find  that  your  current  cost  estimation 
process  relies  heavily  on  expert  judgment? 

1.  Yes 

2.  No 

3.  Not  Sure 
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Experts  Tend  to  Be  Over-Confident 


Most  people  are  significantly  overconfident 
about  their  estimates,  especially  educated 
professionals 


90%  Confidence 
Interval 


(AIE  =  Hubbard  Generic  Calibration  Training) 


Group 

Subject 

%  Correct  (target  90%) 

Harvard  MBAs 

General  Trivia 

40% 

Chemical  Co.  Employees 

General  Industry 

50% 

Chemical  Co.  Employees 

Company-Specific 

48% 

Computer  Co.  Managers 

General  Business 

17% 

Computer  Co.  Managers 

Company-Specific 

36% 

Used  with  permission  from  Douglas  Hubbard  Copyright  HDR  2008  dwhubbard@hubbardresearch.com 


QUELCE 

Carne^e  Mellon  Twitter  #seiwebin„. 

^  ©  2012  Carnegie  Mellon  University 


Software  Engineering  Institute 


Percent  Correct  of  all  Candidates  for  all  Tests 
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Experiments  confirm  that  calibrated  judgment  can  be  taught. 
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Future  Research  Activities 
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Create  A  Repository  for  Quantifying  Program  Execution 
Uncertainties 


Subject  Matter  Experts  need  DoD 
MDAP  data  about  uncertainty  to 
quantify  relationships  of  program 
change  drivers  and  their  impact  on 
program  execution. 

Why  Hard?  Empirical  data  need 
to  be  identified,  accessed, 
extracted  and  analyzed  from  a 
myriad  of  sources.  Data  about 
program  change  is  not  structured 
nor  quantified  for  use  in 
estimation. 

DoD  Need:  Quantified  information 
about  cost  driver  uncertainty 
should  inform  estimates. 
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Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


1.  Identify 
Change 
Drivers  & 
States 

2.  Reduce  Cause  and 
Effect  Relationships 
via  Design  Structure 
Matrix  techniques 

3.  Assign 
Conditional 
Probabilities  to 
BBN  Model 

-> 

4.  Calculate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 

■> 

5.  Monte  Carlo 
Simulation  to 
Compute  Cost 
Distribution 

Information  Cloud 


Program  Rpts: 

SARS,  DAES 

DoD 

Repositories 

Program  Artifacts: 

ARJ 

Experts 


MDAP  Data 
Sources 


CAPE  and 
Service  Cost 
Centers 
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Repository:  Analyze  Existing  Data  to  Model  Program 

Execution  Uncertainties  - 1 


Solution 


Program  Change 


Prog 

State 

Driver 

DDG51 

cond  1 

CONORS 

cond  2 

System 

cond  3 

CapDef 

JTRS 

cond  1 

InterOp 

cond  2 

Prod  uc 

F22 

cond  1 

Contract 

cond  2 

Function 

cond  3 

CONORS 

Materiel  Solution  Analysis  Phase  -  Pre  Miiestone  Estimate 


4.  Calculate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 

1.  Identify 
Change 
Drivers  & 
States 

2.  Reduce  Cause  and 
Effect  Relationships 
via  Dependency 
Structure  Matrix 
techniques 

3.  Assign 
Conditional 
Probabilities  to 
BBN  Model 

5.  Monte  Carlo 
Simulation  to 
Compute  Cost 
Distribution 

For  C2  systems, 
how  often  does 
Strategic  Vision 
change? 


(J^A — ~ 


Records  show  that  Strategic 
Vision  changed  in  45%  of  the 
MDAPS 


Driver  State  Matrix 


The  Materiei  Soiution  of  a 

global  network  command  and 
control  system  anticipates  a 
possible  change  in  Strategic 
Vision  which  will  include 
allied  participation. 

Sharing  information  with  allies 
creates  new  encryption 
requirements  (a  change  in 

Mission/CONOPs). 

These  changes  lead  to 
changes  in  Capability 


Repository  identifies  probability 
of  change  in  MDAP  cost  drivers. 
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Repository:  Analyze  Existing  Data  to  Model  Program 

Execution  Uncertainties  -  2 


Solution 


Program  Change 
Repositor 


Prog 


DDG51 


cond  1 


cond  2 


cond  3 


JTRS 


F22 


State 


cond  1 


cond  2 


cond  1 


cond  2 


Driver 
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System  De 
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Contract 
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Materiel  Solution  Analysis  Phase  -  Pre  Milestone  Estimate 


1.  Identify 
Change 
Drivers  & 
States 


2.  Reduce  Cause  and 
Effect  Reiationships 
via  Dependency 
Structure  Matrix 
techniques 


3.  Assign 
Conditionai 
Probabiiities  to 
BBN  Modei 


4.  Caicuiate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 


5.  Monte  Cario 
Simuiation  to 
Compute  Cost 
Distribution 


If  strategic  Vision 
changes,  what 
else  change^ 


> 


70%  of  the  time  the 
Mission/CONOPS  changes 


The  Materiel  Solution  of  a 

global  network  command  and 
control  system  anticipates  a 
possible  change  in  Strategic 
Vision  which  will  include 
allied  participation. 

Sharing  information  with  allies 
creates  new  encryption 
requirements  (a  change  in 

Mission/CONOPs). 


Driver  State  Matrix 


Repository  identifies 
cascading  effects  of 
change  in  MDAP  cost 
drivers. 
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Repository:  Analyze  Existing  Data  to  Model  Program 

Execution  Uncertainties  -  3 


Solution 


Materiel  Solution  Analysis  Phase  -  Pre  Miiestone  Estimate 


4.  Caicuiate  Cost 
Factor 

Distributions  for 
Program  Execution 
Scenarios 

1.  Identify 
Change 
Drivers  & 
States 

2.  Reduce  Cause  and 
Effect  Reiationships 
via  Dependency 
Structure  Matrix 
techniques 

3.  Assign 
Conditionai 
Probabiiities  to 
BBN  Modei 

5.  Monte  Cario 
Simuiation  to 
Compute  Cost 
Distribution 

When 

both  Strategic  Vision  &  Mission/CONOPs 
experience  change,  the  BBN  caicuiates  that 
Capabiiity  Definition  wiii  aiso  change 
95%  of  the  time. 


Dr 


Joint  Conditional 
Probabilities  can  be 
calculated  for 
downstream  changes. 


nx 


BBN  Model 


Software  Engineering  Institute  Carnegie  Mellon 


The  Materiel  Solution  of  a 

global  network  command  and 
control  system  anticipates  a 
possible  change  in  Strategic 
Vision  which  will  include 
allied  participation. 

Sharing  information  with  allies 
creates  new  encryption 
requirements  (a  change  in 

Mission/CONOPs). 

These  changes  lead  to 
changes  in  Capability 
Definition. 
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QUELCE  Summary 

QUELCE  includes  the  effects  of  uncertainty  in  the  resulting  estimate  by: 

•  Making  visible  the  quantified  uncertainties  that  exist  in  basic  assumptions. 

•  Calculating  uncertainty  of  the  input  factors  to  the  model  rather  than  adjusting 
the  output  factors. 

•  Using  scenario  planning  to  calculate  how  specific  changes  might  affect 
outcomes. 

The  method  utilizes  subjective  and  objective  data  as  input 

•  Historical  data  can  be  used  to  populate  the  BBN  nodes  and  establish  the 
connections  between  the  BBN  and  cost  model  inputs. 

•  Expert  judgments  are  documented  and  made  explicit. 

•  Information  typically  not  used  for  estimation  purposes  can  be  leveraged. 

The  method  explicitly  includes  factors  that  have  been  documented  as 
sources  of  program  failure  in  the  past  but  are  not  typically  captured  by 
cost  models 
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For  More  Information 

QUELCE  Technical  Report: 

http://www.sei.cmu.edu/librarv/abstracts/re 

ports/11tr026.cfm 

SEI  Blog 

http://bloa.sei.cmu.edu 

•  “Improving  the  Accuracy  of  Early  Cost 
Estimates  for  Software-Reliant 
Systems,  First  in  a  Two-Part  Series” 

•  “A  New  Approach  for  Developing  Cost 
Estimates  in  Software  Reliant  Systems, 
Second  in  a  Two-Part  Series” 

•  “Quantifying  Uncertainty  in  Early 
Lifecycle  Cost  Estimation  (QUELCE): 
An  Update” 

Journal  of  Software  Technology 

http://lournal.thedacs.com/issue/64/207 

•  “An  Innovative  Approach  to  Quantifying 
Uncertainty  in  Early  Lifecycle  Cost 
Estimation” 
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Contact  Information 


Presenters  /  Points  of  Contact 

SEMA  Cost  Estimation  Research 
Group 

Robert  Ferguson 

rwf(3>.sei.  emu,  edu 

Dennis  Goldenson 
da@sei.cmu.edu 

Jim  McCuriey 
imccurle@sei.cmu.edu 

Robert  Stoddard 

rws&.sei.  emu,  edu 

Dave  Zubrow 

dz@sei.cmu.edu 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612,  USA 

Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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This  material  is  based  upon  work  supported  by  the  Department  of  Defense  under  Contract  No.  FA8721-05-C-0003  with  Carnegie 
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NO  WARRANTY 
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BS  Join  the  Measurement  and  Analysis  Forum  on  Linkedin  Bf| 


EHIBI 

Measurement  and  Analysis  Forum 

1 

Discussions  Members  Promotions  Jobs  Search  Manage  More... 

Share  group 

http://www.linkedin.com/qroups/Measurement-Analvsis-Forum-2758144 
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